\begin{table*}[t]
\centering \fontsize{8.4}{10.1}\selectfont \setlength{\tabcolsep}{0.5em}
\begin{tabular}{lrrrrrrrrrr}

\toprule

&& \multicolumn{2}{c}{\textbf{Single Sentence}} & \multicolumn{3}{c}{\textbf{Similarity and Paraphrase}} & \multicolumn{4}{c}{\textbf{Natural Language Inference}} \\
% \cline{3-4}
% \cline{5-7}
% \cline{8-11} \\ % TODO: The spacing doesn't work well here for some reason.
\textbf{Model} & \multicolumn{1}{c}{\textbf{Avg}} & \multicolumn{1}{c}{\textbf{CoLA}} & \multicolumn{1}{c}{\textbf{SST-2}} & \multicolumn{1}{c}{\textbf{MRPC}} & \multicolumn{1}{c}{\textbf{QQP}} & \multicolumn{1}{c}{\textbf{STS-B}} & \multicolumn{1}{c}{\textbf{MNLI}} & \multicolumn{1}{c}{\textbf{QNLI}} & \multicolumn{1}{c}{\textbf{RTE}} & \multicolumn{1}{c}{\textbf{WNLI}} \\
\midrule
\multicolumn{11}{c}{Single-Task Training} \\
\midrule

BiLSTM & 63.9 & 15.7 & 85.9 & 69.3/79.4 & 81.7/61.4 & 66.0/62.8 & 70.3/70.8 & 75.7 & 52.8 & \textbf{\underline{65.1}} \\

~~+ELMo & 66.4 & \textbf{\underline{35.0}} & \underline{90.2} & 69.0/80.8 & 85.7/65.6 & 64.0/60.2 & 72.9/73.4 & 71.7 & 50.1 & \textbf{\underline{65.1}} \\

~~+CoVe & 64.0 & 14.5 & 88.5 & \underline{73.4}/\underline{81.4} & 83.3/59.4 & \underline{67.2}/\underline{64.1} & 64.5/64.8 & 75.4 & \underline{53.5} & \textbf{\underline{65.1}} \\

~~+Attn & 63.9 & 15.7 & 85.9 & 68.5/80.3 & 83.5/62.9 & 59.3/55.8 & 74.2/73.8 & \underline{77.2} & 51.9 & \textbf{\underline{65.1}} \\

~~+Attn, ELMo & \underline{66.5} & \textbf{\underline{35.0}} & \underline{90.2} & 68.8/80.2 & \textbf{\underline{86.5}}/\textbf{\underline{66.1}} & 55.5/52.5 & \textbf{\underline{76.9}}/\textbf{\underline{76.7}} & 76.7 & 50.4 & \textbf{\underline{65.1}} \\

~~+Attn, CoVe & 63.2 & 14.5 & 88.5 & 68.6/79.7 & 84.1/60.1 & 57.2/53.6 & 71.6/71.5 & 74.5 & 52.7 & \textbf{\underline{65.1}} \\

\midrule
\multicolumn{11}{c}{Multi-Task Training} \\
\midrule

%BiLSTM & 63.5 & 24.0 & 85.8 & 71.9/82.1 & 80.2/59.1 & 68.8/67.0 & 65.8/66.0 & 71.1 & 46.8 & 63.7 \\
BiLSTM & 62.2 & 11.6 & 82.8 & 74.3/81.8 & 84.2/62.5 & 70.3/67.8 & 65.4/66.1 & 74.6 & 57.4 & \textbf{\underline{65.1}} \\

%~~+ELMo & 64.8 & \underline{27.5} & 89.6 & 76.2/83.5 & 78.5/57.8 & 67.0/65.9 & 67.1/68.0 & 66.7 & 55.7 & 62.3 \\
~~+ELMo & 67.7 & 32.1 & 89.3 & 78.0/\textbf{\underline{84.7}} & 82.6/61.1 & 67.2/67.9 & 70.3/67.8 & 75.5 & 57.4 & \textbf{\underline{65.1}} \\

%~~+CoVe & 62.2 & 16.2 & 84.3 & 71.8/80.0 & 82.0/59.1 & 68.0/67.1 & 65.3/65.9 & 70.4 & 44.2 & \textbf{\underline{65.1}} \\
~~+CoVe & 62.9 & 18.5 & 81.9 & 71.5/78.7 & \underline{84.9}/60.1 & 64.4/62.7 & 65.4/65.7 & 70.8 & 54.7 & \textbf{\underline{65.1}} \\

%~~+Attn & 65.7 & 0.0 & 85.0 & 75.1/\textbf{\underline{83.7}} & 84.3/63.6 & \underline{73.9}/\underline{71.8} & 72.2/72.1 & \underline{82.1} & \textbf{\underline{61.7}} & 63.7 \\
~~+Attn & 65.6 & 18.6 & 83.0 & 76.2/83.9 & 82.4/60.1 & 72.8/70.5 & 67.6/68.3 & 74.3 & 58.4 & \textbf{\underline{65.1}} \\

%~~+Attn, ELMo & \textbf{\underline{69.0}} & 18.9 & \textbf{\underline{91.6}} & \textbf{\underline{77.3}}/83.5 & \underline{85.3}/\underline{63.3} & 72.8/71.1 & \underline{75.6}/\underline{75.9} & 81.7 & 61.2 & \textbf{\underline{65.1}} \\
~~+Attn, ELMo & \textbf{\underline{70.0}} & \underline{33.6} & \textbf{\underline{90.4}} & \textbf{\underline{78.0}}/84.4 & 84.3/\underline{63.1} & \underline{74.2}/\underline{72.3} & \underline{74.1}/\underline{74.5} & \textbf{\underline{79.8}} & \underline{58.9} & \textbf{\underline{65.1}} \\

%~~+Attn, CoVe & 64.3 & 19.4 & 83.6 & 75.2/83.0 & 84.9/61.1 & 72.3/71.1 & 69.9/68.7 & 78.9 & 38.3 & \textbf{\underline{65.1}} \\
~~+Attn, CoVe & 63.1 & 8.3 & 80.7 & 71.8/80.0 & 83.4/60.5 & 69.8/68.4 & 68.1/68.6 & 72.9 & 56.0 & \textbf{\underline{65.1}} \\

\midrule
\multicolumn{11}{c}{Pre-Trained Sentence Representation Models} \\
\midrule

CBoW & 58.9 & 0.0 & 80.0 & 73.4/81.5 & 79.1/51.4 & 61.2/58.7 & 56.0/56.4 & 72.1 & 54.1 & \textbf{\underline{65.1}} \\

Skip-Thought & 61.3 & 0.0 & 81.8 & 71.7/80.8 & 82.2/56.4 & 71.8/69.7 & 62.9/62.8 & 72.9 & 53.1 & \textbf{\underline{65.1}} \\

InferSent & 63.9 & 4.5 & \underline{85.1} & 74.1/81.2 & 81.7/59.1 & 75.9/75.3 & 66.1/65.7 & 72.7 & 58.0 & \textbf{\underline{65.1}} \\

DisSent & 62.0 & 4.9 & 83.7 & 74.1/81.7 & 82.6/59.5 & 66.1/64.8 & 58.7/59.1 & 73.9 & 56.4 & \textbf{\underline{65.1}} \\

GenSen & \underline{66.2} & \underline{7.7} & 83.1 & \underline{76.6}/\underline{83.0} & \underline{82.9}/\underline{59.8} & \textbf{\underline{79.3}}/\textbf{\underline{79.2}} & \underline{71.4}/\underline{71.3} & \underline{78.6} & \textbf{\underline{59.2}} & \textbf{\underline{65.1}} \\

\bottomrule
\end{tabular}
\caption{Baseline performance on the GLUE tasks. 
% Bold denotes the best result in each column; underline denotes best result per within a section. 
For MNLI, we report accuracy on the matched and mismatched test sets. For MRPC and Quora, we report accuracy and F1. For STS-B, we report Pearson and Spearman correlation. For CoLA, we report Matthews correlation. For all other tasks we report accuracy. All values are scaled by 100.
% \textit{Avg} is the macro-average of all metrics, weighting tasks equally and first averaging metrics within each task as needed. % SB: We say this already.
A similar table is presented on the online platform. % SB: The appendix shows this.
}
\label{tab:benchmark}
\end{table*}

\section{Benchmark Results}\label{sec:experiments}

We train three runs of each model and evaluate the run with the best macro-average development set performance. For single-task and sentence representation models, we evaluate the best run for each individual task. We present performance on the main benchmark tasks in \autoref{tab:benchmark}. 

% single task baselines
In most cases, using multi-task training over single-task training yields better overall scores, particularly among the parameter-rich attention models.
Attention generally hurts performance in single task training, but helps in multi-task training.
We see a consistent improvement in using ELMo embeddings in place of GloVe or CoVe embeddings, particularly for single-sentence tasks.
Using CoVe slightly improves on GloVe for single task training but not for multi-task training.

% pretrained models
Among the pre-trained sentence representation models, we observe fairly consistent gains moving from CBoW to Skip-Thought to Infersent and GenSen.
% DisSent ranks behind GenSen but fluctuates between Skip-Thought and InferSent in per-task ranking. % SB: Can cut if needed.
Relative to the models trained directly on the GLUE tasks, InferSent is competitive and GenSen outperforms all but the two best.

Looking at results per task, we find that the sentence representation models substantially underperform on CoLA compared to the models directly trained on the task. Similarly, with the exception of InferSent, the sentence representation models are outperformed on SST by our BiLSTM and its non-CoVe variants. 
%These discrepancies indicate a need for better transfer methods for generalizing outside of the tasks a model was trained on and for task diversity in evaluation methods, as we have sought to do with GLUE.
On the other hand, for STS-B, there is a significant gap between the models trained directly on the task and the best sentence representation model.
%, which we interpret as indicating the necessity of using transfer learning methods trained on data outside of the GLUE benchmark in order to solve it.
Finally, there are tasks for which no model does particularly well. On WNLI, no model exceeds most-frequent-class guessing (65.1\%). On RTE and in aggregate, even our best baselines leave room for improvement.
These early results indicate that solving GLUE is beyond the capabilities of current models and methods.
%, and that training on auxiliary tasks seems a necessary and promising direction.

\begin{table*}[t]
\centering \fontsize{8.4}{10.1}\selectfont \setlength{\tabcolsep}{0.5em}
\begin{tabular}{lr@{\hskip 3em}rrrr@{\hskip 3em}rrrrrr}
 \toprule
 && \multicolumn{4}{l}{\textbf{~~Coarse-Grained}} & \multicolumn{6}{c}{\textbf{Fine-Grained}}\\ % SB: Something odd is going on with centering. Fixing manually for now.
\textbf{Model} & \textbf{All} & \textbf{LS} & \textbf{PAS} & \textbf{L} & \textbf{K} & \textbf{UQuant} & \textbf{MNeg} & \textbf{2Neg} & \textbf{Coref} & \textbf{Restr} & \textbf{Down} \\
\midrule
\multicolumn{12}{c}{Single-Task Training} \\
\midrule
BiLSTM & 21 & 25 & 24 & 16 & 16 & 70 & \underline{53} & 4 & 21 & -15 & \textbf{\underline{12}} \\
~~+ELMo & 20 & 20 & 21 & 14 & 17 & 70 & 20 & \textbf{\underline{42}} & 33 & -26 & -3 \\
~~+CoVe & 21 & 19 & 23 & 20 & \underline{18} & 71 & 47 & -1 & 33 & -15 & 8 \\
~~+Attn & 25 & 24 & 30 & 20 & 14 & 50 & 47 & 21 & \textbf{\underline{38}} & -8 & -3 \\
~~+Attn, ELMo & \textbf{\underline{28}} & \textbf{\underline{30}} & \textbf{\underline{35}} & \textbf{\underline{23}} & 14 & \textbf{\underline{85}} & 20 & \textbf{\underline{42}} & 33 & -26 & -3 \\
~~+Attn, CoVe & 24 & 29 & 29 & 18 & 12 & 77 & 50 & 1 & 18 & \underline{-1} & \textbf{\underline{12}} \\

\midrule
\multicolumn{12}{c}{Multi-Task Training} \\
\midrule
BiLSTM & 19 & 16 & 22 & 16 & 17 & 71 & 35 & -8 & 26 & \textbf{\underline{0}} & 8 \\ 
~~+ELMo & 19 & 15 & 21 & 17 & \textbf{\underline{21}} & 70 & \textbf{\underline{60}} & 15 & 26 & \textbf{\underline{0}} & \textbf{\underline{12}} \\
~~+ELMo & 19 & 15 & 21 & 17 & \textbf{\underline{21}} & 70 & \textbf{\underline{60}} & 15 & 26 & \textbf{\underline{0}} & \textbf{\underline{12}} \\
~~+CoVe & 17 & 15 & 21 & 14 & 16 & 50 & 31 & -8 & 25 & -15 & \textbf{\underline{12}} \\
%~~+Attn, -GloVe & 24 & 19 & 30 & 17 & \textbf{22} \\
~~+Attn & \underline{25} & 23 & \underline{32} & \underline{19} & 16 & 58 & 26 & -5 & 28 & -1 & -20 \\
~~+Attn, ELMo & 23 & \underline{24} & 30 & 17 & 13 & \underline{78} & 27 & \underline{37} & 30 & -15 & -20 \\
~~+Attn, CoVe & 20 & 16 & 25 & 15 & 17 & \underline{78} & 37 & 14 & \underline{31} & -15 & 8 \\

\midrule
\multicolumn{12}{c}{Pre-Trained Sentence Representation Models} \\
\midrule
CBoW & 9 & 6 & 13 & 5 & 10 & 3 & 0 & \underline{13} & 28 & \underline{-15} & -11 \\
Skip-Thought & 12 & 2 & 23 & 11 & 9 & 61 & 6 & -2 & \underline{30} & \underline{-15} & 0 \\
InferSent & 18 & 20 & 20 & \underline{15} & 14 & 77 & 50 & -20 & 15 & \underline{-15} & -9 \\
DisSent & 16 & 16 & 19 & 13 & \underline{15} & 70 & 43 & -11 & 20 & -36 & -09 \\
GenSen & \underline{20} & \underline{28} & \underline{26} & 14 & 12 & \underline{78} & \underline{57} & 2 & 21 & \underline{-15} & \textbf{\underline{12}} \\
\bottomrule
\end{tabular}
\caption{Results on the diagnostic set. We report \(R_3\) coefficients between gold and predicted labels, scaled by 100.
The coarse-grained categories are \textit{Lexical Semantics} (\textbf{LS}),
\textit{Predicate-Argument Structure} (\textbf{PAS}),
\textit{Logic} (\textbf{L}),
and \textit{Knowledge and Common Sense} (\textbf{K}). Our example fine-grained categories are \textit{Universal Quantification} (\textbf{UQuant}),
\textit{Morphological Negation} (\textbf{MNeg}),
\textit{Double Negation} (\textbf{2Neg}),
\textit{Anaphora/Coreference} (\textbf{Coref}), \textit{Restrictivity} (\textbf{Restr}), and \textit{Downward Monotone} (\textbf{Down}).}
\label{tab:diagnostic}
\end{table*}

% \begin{figure}[t]
% \centering
% \begin{subfigure}{.9\linewidth}
%   \small
%   \centering
%   \begin{tabular}{ccccc}
%   \toprule
%   \textbf{Gold \textbackslash Prediction} & \textbf{All} & \textbf{E} & \textbf{C} & \textbf{N} \\
%   \midrule
%   \textbf{All} & & 65 & 16 & 19 \\
%   \textbf{E} & 42 & 34 & ~~3 & ~~4 \\
%   \textbf{C} & 23 & 11 & ~~8 & ~~4 \\
%   \textbf{N} & 35 & 19 & ~~5 & 11 \\
%  \bottomrule
% \end{tabular}
% %   \includegraphics[width=.9\linewidth]{figures/confusion}
%   \caption{Confusion matrix for BiLSTM +Attn (percentages).}
%   \label{fig:confusion}
% \end{subfigure}
% SB: We can re-introduce this in the CR.

% \centering
% \begin{subfigure}{.9\linewidth}
%   \small
%   \centering
%   \begin{tabular}{lccc}
%   \\
%   \toprule
%   \textbf{Model} & \textbf{E} & \textbf{C} & \textbf{N} \\
%   \midrule
%   BiLSTM & 71 & 16 & 13 \\
%   $\text{BiLSTM +Attn}^\mathcal{A}$ & \textbf{65} & 16 & \textbf{19} \\
%   BiLSTM +ELMo & 81 & ~~9 & 10 \\
%   BiLSTM +Cove & 75 & 13 & 13 \\
%   \midrule
%   CBoW & 84 & ~~7 & ~~9 \\
%   SkipThought & 80 & ~~8 & 12 \\
%   InferSent & 68 & \textbf{21} & 11 \\
%   DisSent & 73 & 18 & ~~8 \\
%   GenSen & 74 & 15 & 11 \\
%   \midrule
%   Gold & 42 & 23 & 35 \\
%   \bottomrule
%   \end{tabular}
% %   \includegraphics[width=.9\linewidth]{figures/labeldist}
%   \caption{Output class distributions (percentages). Bolded numbers are closest to the gold distribution.}
%   \label{fig:labeldist}
% \end{subfigure}
%   \caption{Partial output of GLUE's error analysis, aggregated across our models.}
%   \label{fig:diagnostic-charts}
% \end{figure}
% SB: We can re-introduce this in the CR.
% JM: sounds good. I'll wait to update the numbers until then.

\section{Analysis}

%By running all of the models on the diagnostic set, we get a breakdown of their performance across a set of modeling-relevant phenomena. 
We analyze the baselines by evaluating each model's MNLI classifier on the diagnostic set to get a better sense of their linguistic capabilities.
Results are presented in \autoref{tab:diagnostic}.

\paragraph{Coarse Categories}
Overall performance is low for all models: The highest total score of 28 still denotes poor absolute performance.
Performance tends to be higher on Predicate-Argument Structure and lower on Knowledge, though numbers are not closely comparable across categories.
%Performance compares similarly between models on the diagnostic dataset as on the main benchmark. 
Unlike on the main benchmark, the multi-task models are almost always outperformed by their single-task counterparts.
%This is perhaps unsurprising because the diagnostic dataset does not prop up multi-task models with small-data tasks like the main benchmark does.
This is perhaps unsurprising, since with our simple multi-task training regime, there is likely some destructive interference between MNLI and the other tasks.
% JM: How about this? I think we should definitely provide an explanation...
%\footnote{This is perhaps unsurprising because the predictions on the diagnostic set were obtained from each model's MNLI classifier, so it is more difficult for multi-task models to get a leg up in comparison to the main benchmark.}. % AW - I think I see what you're saying, but I think it'd take too long to clearly explain what's happening.
The models trained on the GLUE tasks largely outperform the pretrained sentence representation models, with the exception of GenSen.
Using attention has a greater influence on diagnostic scores than using ELMo or CoVe, which we take to indicate that attention is especially important for generalization in NLI.

%One notable trend is the high performance of the BiLSTM+Attn model: though it does not outperform most of the pretrained sentence representation methods (InferSent, DisSent, GenSen) on GLUE's main benchmark tasks, it performs best or competitively on all categories of the diagnostic set.

% JM: can add back in the CR when we add the tables back etc.
%\paragraph{Domain Shift \& Class Priors}
%GLUE's online platform also provides a submitted model's predicted class distributions and confusion matrices. 
%One point is immediately clear: All models severely under-predict \textit{neutral} and over-predict \textit{entailment}. This is perhaps indicative of the models' inability to generalize and adapt to new domains. We hypothesize that they learned to treat high lexical overlap as a strong sign of entailment, and that surgical addition of new information to the hypothesis (as in the case of neutral instances in the diagnostic set) might go unnoticed.
%Indeed, the attention-based model seems more sensitive to the neutral class, and is perhaps better at detecting small sets of unaligned tokens because it explicitly tries to model these alignments.
% Future training and data collection methods might want to account for this bias to guide their models to do deeper reasoning.

%\paragraph{Linguistic Phenomena}
\paragraph{Fine-Grained Subcategories}
%Beyond overall performance numbers, we can gain a better understanding of the models' capabilities by drilling down into the fine-grained subcategories.
%We gain further insight into model behavior by analyzing performance on specific linguistic phenomena.
%The online platform reports scores for each subcategory; we present a few highlights in \autoref{tab:diagnostic}.

%The Universal Quantification category appears easy for most models. 
%Looking at examples, it seems that when universal quantification as a phenomenon is isolated, 
Most models handle universal quantification relatively well.
Looking at relevant examples, it seems that relying on lexical cues such as ``all'' often suffices for good performance.
%Morphological negation examples are similar, where lexical cues often provide good signal.
Similarly, lexical cues often provide good signal in morphological negation examples.

%On the other hand, several categories seem to be adversarially difficult for several models.
We observe varying weaknesses between models.
Double negation is especially difficult for the GLUE-trained models that only use GloVe embeddings. 
This is ameliorated by ELMo, and to some degree CoVe.
%, perhaps because the translation and language modeling objectives teach models that phrases like ``not bad'' and ``okay'' have similar distributions.
Also, while attention improves overall results, attention models tend to struggle with downward monotonicity.
Examining their predictions, we found that the models were sensitive to hypernym/hyponym substitutions as signals of entailment, but predicted it in the wrong direction (as if the substituted word was in an upward monotone context).
Restrictivity examples, which often depend on nuances of quantifier scope, are especially difficult for all models.

Overall, there is evidence that going beyond sentence-to-vector representations, e.g. with an attention mechanism, might aid performance on out-of-domain data, and that transfer methods like ELMo and CoVe encode linguistic information specific to their supervision signal.
However, increased representational capacity may lead to overfitting, such as the failure of attention models in downward monotone contexts.
%Our platform and diagnostic dataset should support future inquiries into these issues, so we can better understand our models' generalization behavior and the kind of information they encode.
We expect that our platform and diagnostic dataset will be useful for similar analyses in the future, so that model designers can better understand their models' generalization behavior and implicit knowledge. 